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INTRODUCTION 

NASA and the U.S. Air Force are working on a joint project to develop a new 
hydrogen- fueled, full-flow, staged combustion rocket engine [1]. The initial testing and 
modeling work for the Integrated Powerhead Demonstrator (IPD) project is being 
performed by NASA Marshall and Stennis Space Centers. A key factor in the testing of 
this engine is the ability to predict and measure the transient fluid flow during engine 
start and shutdown phases of operation. A model built by NASA Marshall in the ROCket 
Engine Transient Simulation (ROCETS) program is used to predict transient engine fluid 
flows. The model is initially calibrated to data from previous tests on the Stennis El test 
stand. The model is then used to predict the next run. Data from this run can then be 
used to recalibrate the model providing a tool to guide the test program in incremental 
steps to reduce the risk to the prototype engine. In our paper, we define this type of 
model as a calibrated model. 

This paper proposes a method to estimate the uncertainty of a model calibrated to 
a set of experimental test data. The method is similar to that used in the calibration of 
experiment instrumentation. For the IPD example used in this paper, the model 
uncertainty is determined for both LOX and LH flow rates using previous data. The 
successful use of this model is then demonstrated to predict another similar test run 
within the uncertainty bounds. The paper summarizes the uncertainty methodology when 
a model is continually recalibrated with new test data. The methodology is general and 
can be applied to other calibrated models. 

BACKGROUND 

In general, modeling uncertainty exists due to numerical accuracy and simplifying 
assumptions and to variations in design conditions, input parameters, and other 
components of a model. Most of the literature on modeling uncertainty has addressed the 
effect of input parameter uncertainty. Some recent work has addressed the area of 
verification and validation (V&V) in an attempt to estimate the other components of 
model uncertainty. 

The basis for modeling uncertainty was adapted from the widely used 
experimental uncertainty analysis. The experimental uncertainty analysis references for 
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consisting of a transducer (t) and associated meter (m). Prior to calibration, the 
uncertainty of the instrumentation system (only considering the effects of the transducer 
and the meter, no measurement process effects) would be the root-sum-square of the 
transducer uncertainty and the meter uncertainty 

u i, = V U ^ +u m (1) 

where u is the standard uncertainty, an estimate of the standard deviation of the parent 
population of the error for each quantity. In the calibration process, a correction factor is 
added to the meter reading to make it match the value from the calibration standard 
system. This correction essentially replaces the uncertainty of the instrumentation system 
with the calibration (c) system uncertainty so that 

U I, =u c (2) 

The calibration system uncertainty will be the root-sum-square of the systematic (fixed) 
standard uncertainty of the calibration standard, b c the systematic standard uncertainty 

of the calibration standard meter, h , (if a different meter is used for the standard from 

s 

that used for the transducer), and any random standard uncertainty in the calibration 
process, s c , so that 


u c=^/ b ?, +b? 2 +Sc (3) 

The systematic standard uncertainty b is a standard deviation level estimate of the 
possible distribution of fixed errors, and the random standard uncertainty is the standard 
deviation of the calibration correction [5]. If the same meter is used for the calibration 
standard and the transducer, then the h term would not be included in Equation 3. 

Usually, a calibration is done over a range, and a curve-fit (cf) is performed to 
develop a relationship between the meter reading and the calibrated output value. The 
curve-fit will not give an exact correction for all meter readings; therefore, there will be 
an additional uncertainty in the calibrated instmmentation system output because of the 
curve-fit error. Often, the effects of this error are estimated as a standard error of 
regression for the curve-fit [4], The instrumentation uncertainty is then 

Ui 3 =V U C+Ucf (4) 

The uncertainty Ut takes into account the calibration correction for the 

A 3 

transducer and the meter and includes the curve-fit uncertainty. In some cases, the meter 
used in testing (m 2 ) is different from the meter used in calibration (mi). Changing the 
meter requires that the systematic standard uncertainties of both meters be included in the 
instmmentation system uncertainty 
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a standard error of regression, or it can be an uncertainty estimate that varies over the 
range of the simulation results, whichever is appropriate. For the calibrated model, the 
simulation uncertainty in Equation 6 is replaced by the root-sum-square of the data 
uncertainty, u d , and the comparison error uncertainty, ue, so that 

u s, = V u d + U E ( 8 ) 

The uncertainty in Equation 8 would apply for simulation results over the range of 
the input parameter values used in the calibration of the model with the assumption that 
the input parameters have the same uncertainties that they had in the calibration process 
(the same transducers are used for input readings or the same source is used for input 
data). Often, the input parameters used in a simulation calculation from a calibrated 
model will be from different sources or transducers from those used in the calibration 
process. These different sources will also have associated uncertainties that are 
independent of the original parameter uncertainties. This case is similar to changing the 
meter in the instrumentation calibration, and the uncertainties from both sets of input 
parameters would have to be included in the simulation uncertainty 

U Sj =7 U d +U E +U sP, +U sp, (9) 

The uncertainties u sp and U S p would be evaluated using Equation 7 taking into account 

only the parameters that changed sources between the calibration and the calculation of 
the new simulation result. 

The development above is for the general case of a calibrated model. In the next 
section, a specific example is given for the IPD testing being done by NASA and the U.S. 
Air Force. 

IPD EXAMPLE 

Experimental Data 

The experimental data used in this study was activation data. Activation is the 
process through which a facility is tested and checked for problems before a test article is 
installed. The facility included the liquid oxygen (LOX) and liquid hydrogen (LH2) run 
tanks to supply the test liquids and the supply piping from the tanks to the test article 
interface. The supply piping included the venturis to measure the mass flow rates and 
many other purge valves and bleed valves. A pneumatically controlled butterfly valve 
was used to represent the IPD test article at the test article interface during the activation 
tests. The run tanks from which the test liquids flowed kept a near constant pressure 
throughout each test. Therefore, the butterfly valves controlled the flow for each system. 

The parameters of interest for this study are the LOX system mass flow rate and 
the LH2 system mass flow rate of the IPD activation tests. Both of these flow rates are 
measured by their respective venturi flow meters. The venturi flow rate, WPUMP, used 
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Table 1. Input Parameter Uncertainty Summary 


Parameter 

Units 

LOX Uncertainty 

LH2 Uncertainty 

c d 

NA 

.00488 

.00484 

F 


negligible 

negligible 

Dt 


.001 inches 

.001 inches 

D, 

inches 

.01 inches 

.01 inches 

P 


1.0% 1 

1.0% 

AP 


1.21 psi 

1.21 psi 


Table 2. Venturi Mass Flow Rate Uncertainty Summary 



■ R 

LOX Flow 
Rate 

Uncertainty 

(%) 

LH2 Flow 
Rate 

Uncertainty 

(%) 

.76+ 

1.70 

.93+ 

2.00 

.89+ 

2.80 

.68 -.76 

1.90 

.84 -.93 

2.10 

.78 -.89 

3.30 

.59 -.68 

2.10 

.76 -.84 

2.40 

.67 -.78 


.51 -.59 

2.40 

.68 -.76 

2.60 

.55 -.67 

4.40 

.42 -.51 

2.90 

.59 -.68 


.44 -.55 

5.50 

.34 -.42 

3.50 

.51 -.59 

3.50 

.33 -.44 

7.80 

.25 -.34 

WMMMM 

.42 -.51 

4.10 

.22 -.33 

11.1 

.17 -.25 



5.10 

.11 -.22 

22.2 

.12 -.17 

10.1 

.25 -.34 

6.80 

.09 -.11 

29.9 

.10 -.17 

11.8 




37.7 



.15 -.17 

■m 

.04 -.07 

55.5 





.02 -.04 

110.9 


A schematic of the IPD activation system modeled in this study is given as Figure 
1 . The overall ROCETS scheme is an iterative, multi-variable, Newton-Rahpson 
predictor-corrector equation solver. It starts with initial pressure and enthalpy guesses at 
the nodes to determine all needed fluid properties. Properties are defined at nodes 
(circles in Figure 1), not in the legs connecting nodes. Then ROCETS calculates flows 
and flow derivatives in each leg from the leg’s resistance value and fluid density. Next 
ROCETS predicts new node pressures and enthalpies to conserve mass, momentum, and 
energy within each volume. Newton-Rahpson iteration is performed to solve the set of 
the mass, momentum, and energy equations imbedded in the flow and volume modules 
representing the different system components. Iteration on the corrector continues at 
each time step until convergence to within a preselected tolerance of the variables is 
reached. If convergence is not attained within a preselected number of steps, the time 
step is reduced and iteration begins anew. Pressures, enthalpies, flow rates, etc., are 
determined at each time step in this manner to get to a solution. 
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Simulation Uncertainty 

Since this IPD activation test model is used to predict future test conditions at 
slightly higher flow rates with the same input data measurements, the simulation 
uncertainty is determined using Equation 8. Therefore, the simulation uncertainty at a 
95% confidence level, U s , is determined as 

U s =2^+111 (12) 

The data uncertainty will be the same as the calibration uncertainty given in Equation 3, 
the root-sum-square of the systematic and random standard uncertainties for the data. 

The flow rate data had oscillating flow as seen in Figures 2-5, but there was negligible 
random uncertainty in these IPD activation tests. The systematic uncertainties, 2bd, for 
the flow rate data is given in Table 2; therefore, for these tests Ud will be one-half of these 
tabulated values and will vary with the predicted flow rate. 

For these tests, the difference between the calibrated model values and the data in 
Figures 2-5 was taken as 95% confidence estimates of ue- The comparison uncertainty 
varies with time for each test. The values are smaller during the initial low-flow and final 
high-flow steady-state regions of the test and are larger during the transient regions. 

The calibrated model was used to predict two additional runs, LOX test 9B and 
LH2 test 1 9A. Both of these tests were similar in setup and duration to the calibration 
data. Test 9B used liquid nitrogen in the LOX system while LH2 was again used in the 
LH2 system. The LOX system test durations were approximately 20 seconds, and the 
LH2 system tests were approximately 60 seconds. The start-up region is time 5-7 seconds 
for the LOX system and time 0-7 seconds for the LH2 system. 

The model was run with the simulation uncertainty applied, and the model was 
compared to the data with the data uncertainty included. The plots of the full run for the 
LH2 system are given in Figures 6, 7, and 8. Figure 6 displays the full range of the 
uncertainty for the simulation and data. The flow rate uncertainty was over 100 % before 
the transient section because the AP measurements of the venturi were very low in the 
low-flow range. Plotting the LH2 modeling with a more reasonable scale that does not 
include unrealistic values less than zero gives Figure 7. The LH2 system transient 
section is displayed in Figure 8. The plots of the full run for the LOX system are given in 
Figures 9, 10, and 11. Figure 9 shows that flow rate uncertainty was over 100 % in the 
low-flow range because, as in the LH2 system, the AP measurements of the venturi for 
the LOX system were very low. Figure 10 plots the data with a more reasonable scale 
that does not include unrealistic values less than zero. The transient section is given in 
Figure 11. 

The comparison run LH2 system test data fell within the predicted uncertainty 
bands for the model. The only point for which this was not true was the beginning of the 
high-flow steady-state section of the test. This was due to a phenomenon in the test that 
was not recorded in the data. The difference between the actual and predicted mass flow 
rate in this region of the test was deemed to not have an adverse effect on the operation of 
the engine. Therefore, it was not considered a constraint to engine testing and was not 
investigated further. However, it was suggested that the cause could be from slight 
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Figure 8. LH2 Test 19A Transient Model/Measured Comparison with Uncertainty 
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Figure 9. LOX Test 9B Model/Measured Comparison with Uncertainty Full Range 
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The comparison run LOX system test data did not fall within the predicted 
uncertainty bands for the model as well as the LH2 data did. The data covers the lower 
uncertainty band shown in the steady state section of the test. This problem was most 
likely due to LN being used as the test liquid instead of LOX. The model was calibrated 
with LOX as the fluid, and the comparison model used LN as the fluid with no changes 
made to the model to account for fluid viscosity changes which would produce leg 
resistance changes. This caused a higher simulation flow rate than was measured. 
However, the general shape of the test data was matched. It is expected that the 
comparison would have been better if LOX had been used as the test liquid. Although, 
even with the different fluid, the difference between the simulation results and the data 
for the high-flow steady-state region was about 8 % as compared to the simulation 
uncertainty of 5 %. Also, this difference should not be a problem for calibrated modeling 
since ROCETS models are only calibrated with different fluids in the early, low-flow rate 
stage of development and not in the later stages which require low simulation 
uncertainties. 

CONCLUSIONS 

A method to determine the simulation uncertainty of a calibrated model was 
developed. The Coleman and Stem [13] model validation process was adapted to help 
formulate the simulation uncertainty methodology required. This methodology was then 
applied to the model when it was used to predict additional test results. 

The uncertainty analysis of the experimental data was done using methodology 
described in Coleman and Steele [5]. The uncertainty of the experimental data was 
essentially the uncertainty of the venturi flow meters used to measure the mass flow rate. 
This uncertainty was dominated by the AP measurement of the venturi. The uncertainty 
of the venturi was acceptable during the transient and high flow rate steady-state regions 
of the tests. However, at low flow rates before the transient section began, the 
uncertainty was over 100 %. This was due to the AP measurement being very low. To 
correct this problem at the lower flow rates a pressure transducer with a smaller range 
would be needed [14]. 

Comparison error between the calibrated model and the calibration data was used 
as a way to quantify the comparison uncertainty associated with the simulation. These 
uncertainties in conjunction with the venturi uncertainty were the two components of the 
overall simulation uncertainty. The comparison uncertainty was large during the 
transient regions and small during the steady-state regions. 

The process of calibrating a model each time new data is available is logical for 
the work that is done at NASA Stennis test facilities. Due to the fact that no two engine 
test series are the same, making one model to encompass all possibilities is impractical. 
The model uncertainty methodology in this study fits this process of a calibrated model 
that is constantly “tweaked.” This tweaking leaves the user with a calibrated model that 
can be used to predict the next test condition. A comparison of the model prediction and 
the next run then shows if additional corrections (calibration) need to be made and how 
reliable the model predictions should be for future safe engine operation. 
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